Making Changes

Every change to the servers starts as a change to the infrastructure-automation repo. Then someone runs the matching make target. Nothing redeploys automatically when the repo changes, so a merged change only takes effect once the target runs.

Who can run make

Admin-only

Every make target changes production directly. There's no staging environment. Only admins run them. This applies to every make command in these docs, whether or not the page repeats it.

If you're not an admin:

1. Create a branch in infrastructure-automation with your change: a playbook, a compose file, an nginx vhost, a .tf file, the Makefile, and so on.
2. Open a pull request on Gitea and ask an admin to review it.
3. Once it's approved, the admin merges it and runs the target.

Always open a pull request, even for a small change. The diff shows the admin exactly what's about to run against production.

Common changes

Change	Edit	Then run
Add or change a public site	networking/nginx-config/<hostname>	make update-webserver, and add the DNS record in Cloudflare
Add a web app	See Adding an App	make deploy-services-user-apps or deploy-services-staff-apps
Change an app's compose file or environment	ansible/hetzner-main/services-*-apps/<app>/	make deploy-services-*-apps
Change an app's secrets	The <app> .env Secure Note in Vaultwarden	make deploy-services-*-apps
Change API gateway settings	ansible/hetzner-main/vars/api-gateway.yml	make update-api-gateway
Install software on a server	That server's playbook	Its update-* target (or setup-*, if the task is tagged setup only)
Change a Grafana dashboard	grafana-dashboards/<name>.json	make update-observability-secops
Resize a VM	On the Proxmox host, then the .tf file	See Changing a VM's size

Opening a port between servers

Find the UFW tasks in the playbook of the server that receives the traffic, and allow the calling server's IP. Follow the existing pattern: SSH from anywhere, plus specific source IPs; everything else denied. On most hetzner-main VMs the UFW tasks are tagged setup only, so run that server's setup-* target for them to apply.

For a new port on Amergin, bind the container to 192.168.100.2 only, then add UFW rules for the callers (10.0.0.14 and/or 10.0.0.17). The existing service playbooks show how.

Remember that UFW doesn't filter ports published by Docker (see Networking → Firewalls). What keeps a port private is the address it binds to.

Adding a VM

1. Terraform: copy an existing .tf file in terraform/hetzner-main/. Pick an unused VMID and IP (the convention is VMID 1xx ↔ IP 10.0.0.xx), and set the CPU, memory and disk. Give the resource a name you won't want to change later.
2. Ansible: add ansible/hetzner-main/setup-<name>.yml, based on the closest existing playbook, and add a group for it in ansible/hetzner-main/inventory.ini. If it should be part of full rebuilds, add it to site.yml too.
3. SSH: add an alias for it to the SSH config file in the Vaultwarden access collection (see Getting Started), and tell the other admins to copy the updated file.
4. Makefile: add setup-<name>, update-<name> and enroll-wazuh-<name> targets, and add them to REAL_TARGETS at the top of the Makefile. Only targets in that list run in the tooling container.
5. Run it:

   make provision PROXMOX_JUMP_HOST=proxmox-host   # creates the VM (read the state warning first)
   make setup-<name>
   make enroll-wazuh-<name>

See Provisioning with Terraform for the warning about the current Terraform state.

Adding a secret

1. Create the item in Vaultwarden, in the right collection.
2. For a password passed to a playbook, add -e <var>="$$(bw get password '<item name>')" to the target in the Makefile.
3. For a whole .env file, use a Secure Note, fetched by the playbook with bw get notes (see Configuring with Ansible → Secrets).

Never commit a secret to the repo, including in vars/ files or compose files.

Removing a server or service

Remove it everywhere it appears, so nothing still points at it:

• the .tf file and the Terraform state (terraform state rm, if the VM is already gone);
• the playbook, its inventory group, site.yml, and its Makefile targets and REAL_TARGETS entries;
• nginx vhosts in networking/nginx-config/, and any UFW rules on other servers that allowed it;
• Alloy and Grafana references, and Uptime Kuma monitors;
• these docs.